Papers with language encoders
Incremental Processing in the Age of Non-Incremental Encoders: An Empirical Assessment of Bidirectional Models for Incremental NLU (2020.emnlp-main)
Copied to clipboard
| Challenge: | a number of languages are processed incrementally, but the best ones do not . we test five models on various datasets and compare their performance using three incremental evaluation metrics. |
| Approach: | They investigate how bidirectional LSTMs and Transformers behave under incremental interfaces . they propose to use bidirectional encoders in incremental mode while retaining non-incremental quality . |
| Outcome: | The proposed models perform better under incremental interfaces than the "omni-directional" BERT model, which achieves better non-incremental performance, but is impacted more by the incremental access. |
Recommend for a Reason: Unlocking the Power of Unsupervised Aspect-Sentiment Co-Extraction (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing review-based recommenders favor large and complex language encoders that can only learn latent and uninterpretable text representations. |
| Approach: | They propose a tightly coupled two-stage approach to extract latent user sentiments and item properties from reviews and an Attention-Property-aware Rating Estimator (APRE). |
| Outcome: | Extensive experiments on seven real-world Amazon review datasets show that the proposed approach extracts the latent user sentiments, item properties, and the complicated interactions between the two components. |
Data or Language Supervision: What Makes CLIP Better than DINO? (2025.findings-emnlp)
Copied to clipboard
| Challenge: | CLIP outperforms self-supervised models like DINO as vision encoders for vision-language models (VLMs) but it remains unclear whether this advantage stems from CLIP’s language supervision or its much larger training data. |
| Approach: | Embedding analysis shows CLIP captures high-level semantics while DINO is more responsive to low-level features like colors and styles. |
| Outcome: | Embedding analysis shows that CLIP captures high-level semantics, while DINO is more responsive to low-level features like colors and styles. |
On Surgical Fine-tuning for Language Encoders (2023.findings-emnlp)
Copied to clipboard
Abhilasha Lodha, Gayatri Belapurkar, Saloni Chalkapurkar, Yuanming Tao, Reshmi Ghosh, Samyadeep Basu, Dmitrii Petrov, Soundararajan Srinivasan
| Challenge: | preserving knowledge of target distribution by fine-tuning all layers can be expensive and may increase data volume requirements. |
| Approach: | They propose an efficient metric based on the diagonal of the Fisher information matrix (FIM score) to select the candidate layers for selective fine-tuning. |
| Outcome: | The proposed metric can select layers leading to strong performance on GLUE and SuperGLUE tasks and across distinct language encoders. |
Can Pre-training help VQA with Lexical Variations? (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Visual Question Answering (VQA) models are closing the gap between oracle performance and model robustness. |
| Approach: | They propose to use language & cross-modal pre-training to investigate the robustness of VQA models towards lexical variations. |
| Outcome: | The proposed model architectures and training techniques improve the performance of the VQA-Rephrasings dataset on rephrased questions. |
On the Robustness of Language Encoders against Grammatical Errors (2020.acl-main)
Copied to clipboard
| Challenge: | Pre-trained language encoders are effective in facilitating downstream natural language processing tasks, but they often assume training and test corpora are clean and it is unclear how the models behave when confronted with noisy input. |
| Approach: | They conduct adversarial attacks to simulate grammatical errors on clean text data. |
| Outcome: | The proposed model performs better when confronted with natural grammatical errors than when faced with noisy input. |
Higher-order Comparisons of Sentence Encoder Representations (D19-1)
Copied to clipboard
| Challenge: | a technique developed by neuroscientists compares activity patterns of different measurement modalities . a recent study examined the correspondence between popular pretrained language encoders and human processing difficulty . |
| Approach: | They employ a technique to compare activity patterns of different measurement modalities . they establish a correspondence between widely-employed pretrained language encoders and human processing difficulty . |
| Outcome: | The proposed technique can be used to compare representational geometries of neural models . it does not require large training samples and is not prone to overfitting, authors say . |